Assignment 3

Importation:

Importing the data:

2. Data description:

About the data:

This dataset contains an airline passenger satisfaction survey. Content

Gender: Gender of the passengers (Female, Male)

Customer Type: The customer type (Loyal customer, disloyal customer)

Age: The actual age of the passengers

Type of Travel: Purpose of the flight of the passengers (Personal Travel, Business Travel)

Class: Travel class in the plane of the passengers (Business, Eco, Eco Plus)

Flight distance: The flight distance of this journey

Inflight wifi service: Satisfaction level of the inflight wifi service (0:Not Applicable;1-5)

Departure/Arrival time convenient: Satisfaction level of Departure/Arrival time convenient

Ease of Online booking: Satisfaction level of online booking

Gate location: Satisfaction level of Gate location

Food and drink: Satisfaction level of Food and drink

Online boarding: Satisfaction level of online boarding

Seat comfort: Satisfaction level of Seat comfort

Inflight entertainment: Satisfaction level of inflight entertainment

On-board service: Satisfaction level of On-board service

Leg room service: Satisfaction level of Leg room service

Baggage handling: Satisfaction level of baggage handling

Check-in service: Satisfaction level of Check-in service

Inflight service: Satisfaction level of inflight service

Cleanliness: Satisfaction level of Cleanliness

Departure Delay in Minutes: Minutes delayed when departure

Arrival Delay in Minutes: Minutes delayed when Arrival

Satisfaction: Airline satisfaction level(Satisfaction, neutral or dissatisfaction)

Graph depicting the ratio between loyal customers and non-loyal customers:

We can see from the graph above that majority of the customers are labeled as loyal customer. Now we will explore how they are divided within the data.

From the sunburst graph we can see that all customers are divided to buisness and personal travel. Each of the groups divided in 3 sub groups of 3 classes: Buiseness, Eco, Eco plus. We can see interesting fact that in Eco class of Business travel, loyal and disloyal customers shares the quantity by 50/50, while in Personal Travel in same Eco class there is almost all passengers are loyal. From this case we can conclude that person in business travel, flying Eco class, has 50% chance to be disloyal customer. On the other hand person flying Eco class with personal travel reason, is 99% loyal customer.

From the 1st graph above we can clearly see that majority of neutral or dissatisfied customers are in Eco class. Because of this reason flying company should give emphasis on that class in order to figure out the reasons and to reduce them. Most satisfied customers are in business class which is not surprising since conditions in this kind of class are much higher then others.

In the 2nd graph we can see that majority of dissastified and satisfied customers are the loyal customers, which is not surprise, because we saw in previous charts that there are more of them then others.

In the 3rd graph we can see that there are almost same quantity of personal and business reason customers which are dissatisfied. On the other side, majority of satisfied customers are those customers who fly because of business travel, which is also not surprising.

Online Booking:

combined with lower corrolated feature: Departure/Arrival Time Convenient , devided by classes.

Online Boarding, Departure/Arrival Time Convenience grouped by Class:

Eco Plus class: Has relativly high rates of dissatisfaction, regardless the Online boarding time. We can see the effect of the Departure/Arrival Time Convenience mainly when it was rated 0, where the satisfaction rates are low.

Business class: Has high rates of satisfaction regardless the Departure/Arrival Time Convenience. Also there are high rates of Online boarding satisfaction.

Eco class: Like buisness class, the number of satisfied passengers is highercompared to number of dissatisfied passengers.

*Also as expected from the corrolation matrix, we can see that low score of Online boarding has low satisfaction scores.

Type of Travel:

combined with lower corrolated feature: Departure/Arrival Time Convenient , devided by classes.

Departure Delay, Arrival Delay grouped by Type of Travel and class:

Personal travel : dissatisfaction is really affected when arrival and departure delay is high.

As expected from the corrolation matrix, the corrolation between Departure Delay, Arrival Delay and sattisfaction is negative - we can see in this bar graph the reason: bigger number of minutes delay is connected to hiher dissetisfaction and vise- versa.

Gender:

As we accpected from the corrolation matrix, there is no difference between females and males satisfaction-wise.

Age:

From age 7-to-38 and from age 61-to-79, quotient of dissatisfied passengers is very high compared to satisfied passengers. On the contrary, in age range 39-60, quotient of satisfied passengers is higher compared to dissatisfied passengers.

3. Data preperation:

Filling missing values in "Arrival Delay in Minutes":

1.Let's see the disterbution:

  1. It can be seen that all values move around a certain average. Therefore in order to fill in the missing data, we will use the average:

Data Exploration:

Connection between satisfaction and different types of services:

Checkin Service, Inflight Service, On-board Service, Leg-room Service:

In each graph satisfied customers scores are distributed as expected: ratings 4 and 5 are the majority of all scores. On the other hand, majority of dissatisfied customers scores are rated also high (3 and 4), hence it is showed that they are not a good indicators for satisfaction.

Inflight service - same as in previous graph, satisfied customers scores distributed as expected, and dissatisfied customers scores are not.

We created corrolation matrix to see which features are most corrolated :

Conclusions from corrolation matrix:

We can see Cleanliness is higly corroleted with the features descrbing "while on flight" charateristics (Food & Drink" , ,Seat Comfort" , ' Infligth enterteinment" . But, not high enough therfore we will not remove non of them .

We can see a 96% corrolation between Arrival delay in Minutes & Departure Delay in minutes. That is quite obvuois why.

We will combine the 2 into 1 feature : "Delay in minutes". The prons of this action: less running time, without sagnificant impact on accuracy.

PCA 3D & 2D :

From first look at the data we can't really inffer anything yet, we will run clustering model to get better insigths from the data

4. Features effecting costumer's satisfaction:

Unsupervised algorithms:

We will use clustering algorithm in order to create features that may help use classidy costumer's satisfaction in our future work. Our goal is to reduce the number of features in the porpuse of lowering the running time without effecting the accuracy.

We can see that the labels mostly don't overlap and therefore we have a good separation of the data points using the KMeans labels.

We will check it with ground truth labels in satisfaction.

From the t-sne plot above we can see that there are 6 groups. The majority of satisfied customers are in 2 big clusters (yellow). Now we will run kmeans to understand what special about this 2 groups and try to understand which features influenced on them.

We got 6 clusters after kmeans execution. As we remember the interesting us clusters are clusters number 1 (dark purple) and number 2(dark pink). We will run get_melted_clusters function to understand what is most important for this two clusters.

In the plots above we can see all clusters features importance. We will focus on clusters 1 and 2 (yellow and green). We analyse each feature and conclude that:

Features that yellow class rated as 4 and 5: Baggage handling, cleanliness, food and drink, inflight entertainment, inflight service, leg room service and seat comfort

Features that green class rated as 4 and 5: Baggage handling, inflight entertainment, inflight service, leg room service and seat comfort.

Feature resembeling:

We noticed from the corrolation matrix a strong corrolation between 'Departure Delay in Minutes' and 'Arrival Delay in Minutes', as expected. We therfore will merge it into 1 feature called :"Delay in minutes" in order to generelaize our training, avoid overfitting and make the algorithms ahed run faster.

From our results of the KMean clustering we saw that :

are evenlly distrebuted among all clusters. Therfore we can conclde those features has low prediction rate, and for the same reasons mantioned above, we will merge those into 1 feature called : "Services off flight"

We can see that there is a lot of similatiry between 2 graphs colors.

5. Maximize preformance for classifaying and identifing cosumer's satisfaction:

How are we going to maximize preformance?

Out-of-Fold Predictions for Ensembles:

common use for out-of-fold predictions is to use them in the development of an ensemble model. An ensemble is a machine learning model that combines the predictions from two or more models prepared on the same training dataset.

As Base models: Models evaluated using k-fold cross-validation on the training dataset and all out-of-fold predictions are retained.

  1. RandomForest 2.SVM 3.Adaboost

As Meta-Model: Model that takes the out-of-fold predictions made by one or more models as input and shows how to best combine and correct the predictions.

Gradient Boost

** Out-of-Fold Predictions : Predictions made by the model during the k-fold cross-validation procedure on the holdout examples.

Why does it maximize the preformance?

We are avoiding the possibility for "optimistic bias" because the samples were used in the training of each base model. This optimistic bias means that the predictions will be better than normal, and the meta-model will likely not learn what is required to combine and correct the predictions from the base models.

By using out-of-fold predictions from the base model to train the meta-model, the meta-model can see and harness the expected behavior of each base model when operating on unseen data, as will be the case when the ensemble is used in practice to make predictions on new data.

Finally, each of the base models are trained on the entire training dataset and these final models and the meta-model can be used to make predictions on new data. The performance of this ensemble can be evaluated on a separate holdout test dataset not used during training.

class SklearnHelper :

a class of helper methods in sklearn. We will train and fit all of the algoritms, so in order to make a readble code and won't need to write the same methods five times if we wanted to invoke five different classifiers.

Plots showing presentage of feature importance by each model trained:

Now we will train the kgboost algoritm (our meta-model) using the mean of feature importance from our base model's calculations:

Final meta model Accuracy score:

xgboost: 96%

When running each model on its own, we got good accuracy's (all above 90%) but, nut as good as 96.3%. We can infer that optimization of prediction did occur when we used the Out Of Folds method to create meta model.

(we will show all the other models results later in a table)

Here are the results of all algoritms for comparison:

Regression models with new label:

In the first task we will create new label out of 3 different features and omit them from our data. We chosed 3 different features, each of them belongs to different phase of the fligh: Ease of Online booking, Seat comfort, Baggage handling.

We chose 4 regression models: Linear regression, Random forest, Knn, XGBoost. We will compare between them with next metrics:

  1. R^2 - Coefficient of determination. Goodness of fit. Best possible score is 1.0 and it can be negative.

  2. MSE - Mean Squared Error. Tells how close a regression line to a set of data points.

  3. RMSE - Root Mean Squared Error. It measures the standard deviation of residuals.

  4. MAE - Mean Absolute Error. Represents the average of the absolute difference between the actual and predicted values in the dataset. It measures the average of the residuals in the dataset.

  5. MAPE - Mean Absolute Percentage Error. Measures how accurate a forecast system is. It measures this accuracy as a percentage.

  6. EV - Explained Variance. Measures how far observed values differ from the average of predicted values, i.e., their difference from the predicted value mean.

  7. Max_Error - Shows the maximum error out of all predictions.

Linear regression

Random forest regression

KNN regression

XGBoost

No we will gather all results into one:

From the graph above we can see that the results from all models are quite similar and behave similary. After deeper analysis we can conclude that Knn regressor did the worst job because of higher then others errors, low R^2, and the highest Max_Error. On the other side XGBoost and Random Forest regressors got best results. There is little difference between their results except Max_Error, where Random Forest showed slightly better result then XGBoost.

In next task we will return the omitted features and rerun the models again. In the end we will compare between the results.

Gathering all data together:

From the graphs comparison above we can see how "label features" influence the results. Knn model is the only one that improved slightly, while the other models changed significantly. Linear regression got the highest R^2 and EV scores (1.0) and the lowest errors which can be neglected. Random forest and XGBoost also showed good results - R^2 and EV scores are almost 1.0 and very low errors rate. Random forest stands out with hight Max_error number in comparison to XGBoost.

We conclude that there is an impact on algorithms when data includes columns from which a label was calculated. Best results was shown by linear regression. In our opinion this happened because the label was calculated by linear equation (mean) and hence linear regressor succeded to predict corectly. In addition to our conclusion, linear regressor was the fastest model (less than a second) and in this case of data this model is highly recommended.

Summary:

7. Conclusion from work:

We learned alot from this excrise, we learned about xgboost algoritm and found it very helpul in regression and classification problems (we will elaborate on the model later). We saw that using clustering models can improve our model's preformances - running time wise, without effecting the accuracy results. when selecting feature that are not important for the classification, and modyfiny them (removing, combaibig them together), its a good practice to do before classifying tasks.

We got to learn about model improvment using other model's predictions (section 5).

Our prime problem ws with the data size. It was large and required alot of time to train our models. We accualy tried neural nets model for section 5 and gave it up for its long running time. Also, we found several solutions to deal with the long running time problem. For example, in the SVM model, we used parameters that help in this section- c<1 & using only the linear kernel, which according to the internet, is the fastest one. Moreover, the stantartization of the data was key for not only redue the running time of our models (espacially in models based on distance calculation such as knn and svm), it was also crutial for the accuracy and visualization. We sampled from our data, to represent some of the point after dimantion reduction in our PCA visualization.

Final results classification models:

We can see the accurecies are quite good.

The beat accuracy is for our gradient boost after preforming out of fold geature importance inhancing. We used the feature importance mean from all other classifiers results as a new feature for the xgboost algoritm. As we see in the results, the algorithm's preformance was indeed improved from 93% to 96% accuracy.

Other models where good as well, especially the random forest model. But the best is the Xgboost after preforming the procedure

Explanation for classification results:

In all models we tested them after feature improvments in section 4.

Random Forest :

We used random forest and got good accuracy. We had a good results becuasee the problem we dealed with need classification model and it's about a determination that can be solved perfectly with a trees model and random forest can pick the best tree that gives the best results. Random forest is an algorithms that have built-in feature selection methods. We calculate feature importance using node impurities in each decision tree. In Random forest, the final feature importance is the average of all decision tree feature importance.

Adaboost :

We used adaboost which is also a forest based model. We got hood results because problem like costumers satisfaction (binary desicion ) can be solved perfectly using tree models .

SVM :

this algorithm try to find the best linear vector between the clusters to define them clearly so in our case we have two clusters 1 and 0 . so when we want to predict about another object ( new one) we can see the devision between the clusters and where it belongs and the vectors always get fixed when new object added according to its place.

Choosing a kernel function is not an easy task. We chose to use only linear kernel for it's the only kernel allowing feature importance calculation. Another reason is SVM takes a lot of time for training with large datasets, using linear kernel as many Google site's suggested, is the fastest kernel.

XGBoost :

XGBoost is one of the most popular machine learning algorithm these days. Regardless of the type of prediction task at hand; regression or classification. XGBoost is well known to provide better solutions than other machine learning algorithms. In fact, since its inception, it has become the "state-of-the-art” machine learning algorithm to deal with structured data. and that's why we got high results this algorithem is one of the best algorithems that are now in use, it can do optimizaton for the feature by it self (accordin to kaggle, youtube and medium)

XGBoost - Meta model:

We used the out-of-fold predictions to create and the meta-model. WE trained each base model on the entire training dataset, then make a prediction for each example in the training dataset and use the predictions as input to the meta-model. The problem is the predictions will be optimistic because the samples were used in the training of each base model. This optimistic bias means that the predictions will be better than normal, and the meta-model will likely not learn what is required to combine and correct the predictions from the base models.

By using out-of-fold predictions from the base model to train the meta-model, the meta-model can see and harness the expected behavior of each base model when operating on unseen data, as will be the case when the ensemble is used in practice to make predictions on new data.

Finally, each of the base models are trained on the entire training dataset and these final models and the meta-model can be used to make predictions on new data. The performance of this ensemble can be evaluated on a separate holdout test dataset not used during training.We used the training data for the base model predictions using kfold method. And the testing data- the "unseen data" ad testing for our Met model.

Regression models without "label features" comparison:

Model $R^2$ MSE RMSE MAE MAPE EV Max_Error .
Linear Regression 64% 0.21 0.468 0.356 0.123 64% 2.54
Random Forest 71.9% 0.174 0.417 0.312 0.107 71.9% 2.27
KNN 45.7% 0.336 0.58 0.454 0.163 46.4% 2.71
XGBoost 71.7% 0.175 0.418 0.312 0.106 71.7% 2.43

Description of regression models:

Linear Regression

Very fast model, executes within a second. In r-squared estimation it got 64% which means that 36% of the data
can not be accounted for by this model. The MSE, RMSE, MAE, and MAPE are worse than other models (except KNN).
In addition maximum error is slightly higher than others but is not significant.
There  were no problems in operating the model, it is worked fast and easy to execute.

Random Forest

This model in r-squared estimation got almost 72%, which is much better than previous model.Estimation of 4 
errors are lower also which means that random forest coped better with the prediction of calculated label.
The main problem with this model at this stage was long time running while fitting it. In order to cut the
time we lowered the number of Cross Validation from 5 to 3, and in other parameters we inputed 3,4 options.

KNN

This model showed itself very bad. Its' r-square estimator showed 45.7% which means that more than half of the
data can not be accounted for by this model. Error rates are the highest out of all compared models which
reflects the low r-square. We didn't face any specific problems with running this algorithm.

XGBoost

This model showed very good results which are similar to random forest. All errors rate are also similar to
random forest with very small difference. In order to cope with time running of the model we reduced number
of variables in grid search and also lowered number of cross validation.

Regression models with "label features" comparison:

Model $R^2$ MSE RMSE MAE MAPE EV Max_Error .
Linear Regression 100% 0 0 0 0 100% 0
Random Forest 99.6% 0.0025 0.05 0.027 0.0101 99.6% 0.733
KNN 62.3% 0.233 0.482 0.372 0.134 63.2% 2.33
XGBoost 99.9% 0.0002 0.014 0.011 0.004 99.9% 0.065

Description of regression models:

Linear Regression

We can see perfect results. 100% r-squared and 0 errors rates. We conclude that because of the existance
of 3 columns which from them was calculated label, the calculation result was recieved after linear
equasiotn (mean), and this fis perfect linear regression model.

Random Forest

If there was a competition then this model would get a 3rd place out of 4 models. But actually its results
are very good and slightly different from the second place algorithm.

KNN

Although this model got better results in all estimators than in previous run, still this results are
far from good. r-square is still low and errors rates are high

XGBoost

This model showed almost perfect results. R-square is almost 100% and errors are very slow.In cases of unknown
data (linear or not) this model is recommended because of his high results.

Thank you for reading

cognitiveclass.ai logo